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The  most  fundamental  thing  to  remember  when  designing  quality  research  is 
that  a  clearly  worded  question  must  be  formulated  before  the  design  can  be 
addressed.  The  question  must  always  determine  the  design,  not  vice  versa.  The 
question  should  be  formulated  in  the  following  simple  form: 

What  is  the  effect  of  "X"  on  ”Y"? 

Where  ”X"  is  the  independent  variable  and  "Y"  is  the  dependent  variable. 

In  situations  with  more  than  one  question,  the  following  points  should  be 
considered : 

1.  Decide  if  each  of  the  questions  is  necessary. 

2.  Consider  asking  the  most  important  question  first. 

3.  Consider  asking  the  most  fundamental  question  first. 

4.  Consider  asking  the  most  simple  or  "doable”  question  first. 

5.  Remember  that  a  series  of  simple  questions  is  usually 
superior  to  a  single  complex  question. 

Once  the  question  has  been  created,  the  design  will  logically  follow.  The 
most  important  task  in  experimental  design  is  to  precisely  measure  the 
independent  and  dependent  variables  while  accounting  for  all  other  variables 
which  could  possibly  influence  the  dependent  variable.  Control  is  a  key  word. 
There  are  different  strategies  for  controlling  the  unwanted  effects  of  nuisance 
variables.  The  methods  listed  in  the  following  paragraphs  are  discussed  in 
detail  by  most  research  design  texts  (e.g.,  Robinson,  1976;  Myers,  1979; 
Kerlinger,  1973;  etc.). 


Independent  Variable  (IV)  If  a  variable  is  one  that  is  of  primary  concern 
in  the  research  (i.e.,  the  variable  whose  effects  are  being  studied),  then  that 
variable  becomes  the  IV.  The  IV  is  always  carefully  controlled  and  manipulated 
in  the  experiment. 

Randomization  (RAND)  This  is  probably  the  most  commonly  used  (and  misused) 
technique  for  controlling  the  effects  of  a  variable.  For  example,  if  one  wants 
to  control  for  the  effects  of  having  troops  from  different  companies,  he/she 
could  randomly  assign  all  troops  to  the  various  experimental  conditions.  The 
problem  here  is  that  one  must  follow  the  established  guidelines  for 
randomization  (i.e.,  insure  that  each  event,  subject,  etc.,  has  equal  chance  of 
being  assigned  to  a  given  condition).  To  accomplish  this,  tables  of  random 
numbers  are  commonly  used.  It  is  important  to  remember  that  randomization  does 
not  necessarily  eliminate  the  problem.  Rather,  randomization  reduces  the 
chances  of  a  nuisance  variable  severely  influencing  the  outcome  of  the 
experiment  to  some  chance  level. 
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Constancy  (CON)  The  unwanted  effects  of  a  variable  can  be  controlled  by 
keeping  chat  variable  constant  for  all  conditions.  For  example,  if  one  is 
concerned  that  different  shooting  lanes  might  affect  scores,  one  could  simply 
require  all  subjects  to  fire  on  the  same  lane.  Two  other  ways  of  achieving 
constancy  with  subject  variables  are  matching  and  blocking.  Matching  is  an 
attempt  to  equate  groups  by  assigning  subjects  to  conditions  on  the  basis  of  how 
they  have  scored  on  a  variable  that  is  known  to  be  highly  correlated  with  the 
dependent  variable.  For  example,  assigning  the  two  best  shooters  to  opposite 
teams,  two  worst  shooters  to  opposite  teams,  and  etc.,  is  an  attempt  to  equate 
the  quality  of  the  two  teams.  (Notice  that  we  are  assuming  that  the  subjects’ 
past  performance,  e.g.,  number  of  hits  in  the  past,  is  correlated  with  their 
performance  in  this  experiment.)  Blocking  is  an  attempt  to  eliminate  some  of 
the  subject  variability  by  placing  subjects  of  similar  backgrounds,  abilities, 
etc.,  in  the  same  groups  (e.g.,  troops  scoring  in  the  top  third  are  placed  in 
one  group,  troops  scoring  in  the  second  third  are  placed  in  another  group,  and 
etc.).  Analysis  of  blocked  designs  becomes  cumbersome  so  it  is  probably  wise  to 
consult  a  design  expert. 

Elimination  (ELIM)  In  order  to  reduce  the  effects  of  an  unwanted  variable, 
one  simple  strategy  is  to  remove  that  variable  from  the  situation.  Obviously, 
if  one  is  concerned  about  the  unpredictable  effects  of  weather  conditions  on 
training  research,  one  could  simply  move  the  training  inside  (if  possible). 

Statistical  Control  (STAT)  There  are  statistical  analyses  that  can 
accomplish  approximately  the  same  thing  as  matching  (see  above).  These  analyses 
attempt  to  equate  the  different  groups  on  the  basis  of  some  correlated  variable 
(called  the  covariate).  This  approach  involves  fairly  sophisticated  statistical 
analyses  and  it  is  probably  best  to  consult  an  expert. 

Correlation  and  Regression  Analysis  (COR)  If  there  are  variables  of  any 
current  or  future  interest  (aside  from  the  independent  and  dependent  variables), 
then  these  variables  should  definitely  be  measured  and  systematically  recorded 
for  analysis.  Either  planned  or  ex  post  facto  correlational  analyses  can  often 
unveil  interesting  and  important  relationships.  One  can  often  economize  by 
"packing"  experimental  investigations  with  correlational  investigations.  It  is 
probably  wise  to  consult  a  design  expert  if  correlation  techniques  are  to  be 
applied  because:  a)  selection  of  appropriate  correlation-regression  analysis 
is  crucial,  b)  there  are  many  potential  pitfalls  in  conducting  such  analyses, 
and  c)  there  are  many  potential  pitfalls  in  interpreting  such  analyses. 

On  the  following  pages  a  matrix  is  provided  that  lists  a  number  of  the  most 
common  variables  that  must  be  dealt  with  in  research  on  small  arms.  This  list 
is  not  comprehensive;  your  situation  may  require  you  to  add  some  variable(s)  to 
this  list.  The  purpose  of  the  matrix  is  to  force  the  user  to  decide  whether 
each  of  these  common  variables  is  relevant  in  this  specific  situation  and  how 
the  user  intends  to  deal  with  it. 
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CHECK  LIST  OF  IMPORTANT  CONSIDERATIONS 


Have  you  considered  safety? 

Have  you  considered  the  time  limitations  for  the  research? 
Have  you  coordinated  with  the  relevant  decision  makers? 

Have  you  coordinated  with  the  relevant  trainers? 

Have  you  coordinated  with  the  subject  population? 

Are  you  personally  able  to  validate  all  measurements? 

What  degree  of  control  do  you  have  in  the  design  stage? 

What  degree  of  control  do  you  have  in  the  data  acquisition 
stage? 

What  degree  of  control  do  you  have  in  the  data  analysis 
stage? 

What  degree  of  control  do  you  have  in  the  reporting  stage? 
Have  you  considered  implementation  problems? 

Have  you  considered  implementation  strategies? 

Were  you  present  during  the  data  collection? 

Did  you  brief  the  trainers? 

Did  you  debrief  the  trainers? 

Did  you  ask  the  trainers  about  problems  or  for  suggestions? 
Did  you  include  the  trainers  in  the  research  process? 

Did  you  ask  the  trainees  about  problems  or  for  suggestions? 
Did  you  triple  check  the  data  analysis? 

Have  you  considered  how  this  system  will  coordinate  with 
other  systems? 

Was  there  a  high  degree  of  cooperation? 


MISCELLANEOUS  POINTS  ON  DESIGN 


1.  Always  try  to  get  validated  pretest  scores  on  all  subjects  under  identical 
conditions.  The  posttest  (collection  of  the  dependent  variable)  will  be  the 
same  for  all  subjects. 


2.  Keep  the  number  of  subjects  in  each  group  equal. 


3.  When  possible  (especially  when  the  number  of  subjects  is  small),  try  to 
obtain  scores  from  all  subjects  in  all  conditions  (this  is  called  repeated 
measures ) . 


4.  In  repeated  measures  designs  either  randomize  the  order  that  the  conditions 
are  presented  or  "counterbalance"  (e.g.,  have  half  the  troops  shoot  the  100 
meter  target  followed  by  the  200  meter  target  and  the  other  half  shoot  the  200 
meter  target  followed  by  the  100  meter  target).  Randomization  and 
counterbalancing  reduce  the  differential  transfer  effects  that  could  occur  as  a 
result  of  repeated  measures. 
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5.  When  two  questions  are  being  asked  simultaneously  (e.g.,  what  is  the  effect 
of  visibility  on  accuracy  and  what  is  the  effect  of  distance  to  the  target  on 
accuracy),  a  matrix  should  be  formed  with  all  levels  of  one  variable  (e.g.,  goo 
or  poor  visibility)  on  one  dimension,  and  all  levels  of  the  second  independent 
variable  (e.g.,  25m,  200m,  and  300m)  as  the  other  dimension: 


Distance 


25m 

200m 

300m 

Visibility  Poor  (F°S> 

Good 

The  above  design  is  called  a  "factorial  design”.  It  is  important  that  (if 
possible),  equal  sized  groups  of  subjects  be  assigned  to  all  possible 
combinations  of  visibility  and  distance  (i.e.,  to  each  cell  in  the  matrix). 
Statistical  analysis  of  a  design  like  this  allows  one  to  answer  at  least  three 
questions : 

1.  What  is  the  effect  of  distance  on  accuracy? 

2.  What  is  the  effect  of  visibility  on  accuracy? 

3.  What  are  the  joint  effects  of  distance  and  visibility  on 
accuracy? 


The  third  question  is  a  product  of  the  factorial  design  that  allows  the  user  to 
analyze  "interactions”.  An  interaction  between  two  variables  means  that  the 
effect  of  one  variable  depends  on  the  level  of  the  second  variable.  For 
example,  an  interaction  is  likely  to  be  found  in  the  above  example.  The  graph 
below  indicates  the  presence  of  an  interaction  (i.e.,  the  effects  of  visibility 
depends  on  distance  from  the  target).  The  hypothetical  results  below  show  that 
there  was  no  effect  of  poor  visibility  (e.g.,  fog)  at  close  range  but  a  very 
large  effect  due  to  fog  at  the  longer  ranges. 


Per  Cent 
Hits 


Distance 
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